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Abstract 


To date there are no evidence-based comprehensive interventions for use in school settings. There are numerous barriers to 
delivery of high-quality interventions in schools that have limited the transfer of research-based interventions to school set- 
tings. Modular Approach to Autism Programing for Schools (MAAPS) is a framework for implementation of evidence-based 
interventions in school settings that is designed to address these barriers. The development and initial evaluation of MAAPS 
was conducted using an implementation-science framework and results indicate that MAAPS is aligned with needs and 
resources available in schools, that it had excellent social validity, and that there is good evidence that MAAPS is effective 
for addressing core and associated features of autism in educational settings. 
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research 


The current reported prevalence of autism spectrum disorder 
(ASD) is estimated at 1 in 59 (Baio et al. 2018). Given that 
many individuals with ASD are served in public schools, 
educators are increasingly responsible for identifying, 
designing, and delivering interventions to address the needs 
of this diverse group of students. Recent reviews of the lit- 
erature have identified both comprehensive and focused 
interventions that address core and associated features of 
ASD. Comprehensive interventions are designed to address 
most or all features of ASD and to be delivered over multi- 
ple months or even years (see Smith and Iadarola 2015 for 
a recent review) whereas focused interventions are limited 
in scope and designed to be implemented for shorter peri- 
ods of time (see Wong et al. 2015 for a review). To date, 
there are no comprehensive interventions with evidence 
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supporting their use in schools. Although a growing body 
of work supports use of focused interventions within school 
settings (Anderson et al. 2017, 2018), they are rarely adopted 
or implemented with fidelity in schools (Hess et al. 2008; 
Stahmer and Aarons 2009; Stahmer et al. 2005). Further, 
even when evidence-based interventions (EBIs) are imple- 
mented, educators tend to use them alongside untested inter- 
ventions or interventions shown to be ineffective (Burns and 
Ysseldyke 2009; Ferreri et al. 2016; Hendricks 2011; Stah- 
mer 2007). 

There are two likely reasons for the lack of adoption or 
sustained use of EBIs. First, most research has been con- 
ducted in highly-controlled settings (e.g., clinics, university- 
run lab schools) by trained research project staff instead of 
typical educators (Kasari and Smith 2013), and often on a 
1:1 basis (Strain and Bovey II 2011). Educators typically 
have little or no training in a specific intervention, fidelity of 
implementation is not monitored in schools, and resources 
to support implementation are limited. Oftentimes 1:1 sup- 
port is not available for most students with ASD. Second, 
the wide diversity in presentation of students with ASD also 
makes adoption challenging as significant knowledge and 
skill is needed to match and customize EBIs to each unique 
student’s needs. Unfortunately for educators, there are no 
evidence-based resources to guide how best to select and 
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combine targeted intervention strategies into a package that 
is both appropriate for any given student and also feasible 
within the educational context. 

Within school-based research, there is growing emphasis 
on using implementation science frameworks to systemati- 
cally assess factors related to uptake of new interventions in 
real world settings (Eccles and Mittman 2006). Of existing 
frameworks, RE-AIM (Glasgow et al. 1999) has particular 
relevance for dissemination efforts in schools (Cook and 
Odom 2013). RE-AIM represents critical implementation- 
related elements (i.e., Reach, Efficacy, Adoption, Imple- 
mentation, and Maintenance) that should be addressed in 
research on an intervention to improve the uptake and sus- 
tainability of that intervention. In this framework, reach is 
the number of individuals who participate in a study and the 
extent to which they are representative of the target popu- 
lation. For example, the large body of research on multi- 
component interventions for young children with ASD may 
not be generalizable to school-aged children. Effectiveness 
refers to not simply whether an intervention was found to 
result in significant improvement in targeted outcomes but 
also documentation of any negative effects or unintended 
consequences. Adoption refers to those (in this case schools) 
who agree to adopt an intervention and the representative- 
ness of those settings and individuals. For example, if an 
intervention is adopted only by private schools for students 
with ASD, we might question whether it would and could 
be adopted within public schools. Implementation refers to 
the extent to which an intervention could be and was imple- 
mented as designed (i.e., with fidelity). Equally important 
is an assessment of the cost of implementation with quality, 
in terms of staff time and other resources needed. Finally, 
maintenance refers to whether and how the intervention 
“sticks,” and becomes part of the organizational practices 
of the classroom, school, or district that participated in the 
study. Although the RE-AIM framework has been evaluated 
in public health research (e.g., Glasgow et al. 1999), and 
various models of implementation science frameworks have 
been examined in exploring the research-practice gap (e.g., 
Dingfelder and Mandell 2011; Locke et al. 2016), RE-AIM 
dimensions have not been consistently included or evaluated 
for classroom-based EBIs focused on students with ASD. 

To address (1) the above-described challenges, and (2) 
the broader implementation context, there is a need for a 
comprehensive intervention approach that provides a frame- 
work for school teams to select and implement focused 
interventions that can address the array of needs presented 
by students with ASD. This framework should provide a 
structure that can enhance feasibility of intervention imple- 
mentation in authentic school environments by having con- 
textual fit and be both effective and sustainable. We devel- 
oped the Modular Approach for Autism Programs in Schools 
(MAAPS) to address these needs. 
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As a first step towards the evaluation of efficacy and adop- 
tion of MAAPS, we assessed the feasibility of MAAPS in 
schools and the practicality and utility of our proposed meas- 
ures. Feasibility studies such as this are a key first step to 
determine whether further evaluation of an intervention in 
a more rigorous clinical trial is appropriate. An interven- 
tion that is appropriate for further study is one that is both 
relevant to the needs the intervention is designed to address 
and has evidence that it will be adopted and sustained. Thus, 
the purpose of our study was to use the RE-AIM framework 
to assess the viability of MAAPS for adoption, sustained and 
high-quality implementation, and achievement of meaning- 
ful outcomes for students with ASD. We assessed this by (1) 
obtaining feedback on MAAPS from experts in the field and 
stakeholders, and (2) conducting a study to assess whether 
outcomes achieved via MAAPS were suggestive of positive 
outcomes for participants. 


Methods 
Phase I: Feedback from Experts and Stakeholders 
Expert Feedback 


Prior to conducting a pilot of MAAPS, we solicited feed- 
back from experts in the field and from stakeholders. From 
experts we hoped to assess the perceived validity of the 
MAAPS framework and of intervention modules. We asked 
stakeholders for feedback on the feasibility and social valid- 
ity of MAAPs. 

We began by soliciting feedback from six researchers 
who had published widely in the areas of ASD and EBP 
and from five practitioners who provided behavioral con- 
sultations and assistance to individuals with ASD and stake- 
holders. Experts completed two structured surveys in which 
item responses were scored on 4-point Likert-scale (Not at 
all, A little, Mostly, Completely). Questions on one survey 
were organized around the overall framework for delivery 
of MAAPS and the other focused on module content. The 
survey on the MAAPS framework consisted of five items 
assessing the extent to which MAAPS aligned with extant 
research, had contextual fit and relevancy for students with 
ASD and their educators, was functional for use by educa- 
tors, and would increase inclusive opportunities. Content- 
specific surveys included items similar to those on the 
framework survey but focused on the specific topic as well 
as additional questions regarding the appropriateness and 
usefulness of the materials. For each survey item, we divided 
the number of experts who rated an item positively by the 
total number of experts. We reviewed all items with a rat- 
ing of less than 0.8 (as recommended by Davis 1992). This 
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feedback informed question development for subsequent 
focus groups. 

Feedback on the framework was positive and resulted in 
no modifications. Feedback on modules also was positive; 
experts provided suggestions for enhancing several modules, 
for example by combining two modules into one and provid- 
ing clearer guidelines for implementation for another. We 
incorporated their feedback in all cases. 


Focus Groups 


To gather information specific to the MAAPS interven- 
tion package, we conducted a series of focus groups with 
school administrators, educators, and parents/guardians. A 
total of 69 people participated across 13 groups, and we 
separated groups based on role (administrator, educator, 
parent). Groups were run by a trained facilitator who used 
a semi-structured interview protocol that was standardized 
across all sites. We audio-recorded the interviews for later 
transcription and scoring. Content analysis of data followed 
standard conventions of team-based qualitative research 
(Guest and MacQueen 2008; Miles et al. 2013; Silverman 
et al. 1990), which included an iterative discussion process 
among research team members as themes and patterns were 
identified for each of the five primary codes and sub-codes. 
All data were gathered before starting content analysis. A 
set of codes based on the focus group questions was iden- 
tified and defined a priori by the research team, followed 
by development of a codebook. The codebook included 
code names, definitions, inclusion/exclusion criteria, and 
code application examples (Miles et al. 2013). The analysis 
phase was conducted at two levels-individual and cross-site. 
First, at least two members of the research team from each 
site independently coded their site’s transcripts and entered 
the coded transcripts into a shared database. Next, several 
virtual cross-site meetings were held to discuss the coding 
results, resolve disagreements, and come to consensus on 
themes. Intercoder reliability (Guest and MacQueen 2008; 
Silverman et al. 1990) was developed via discussions of sim- 
ilarities and differences in coding and coming to agreement 
on the final codes, making any necessary modifications to 
codes and definitions based on the collective understanding 
of the research team members. 


Table 1 data collection 


Not surprisingly, groups identified a lack of resources 
(time, trained personnel, finances) as the primary barrier 
to supporting students with ASD. They also noted a lack 
of training in EBIs as a significant barrier (Iovannone et al. 
2019). With regard to MAAPS, participants’ statements 
suggested that MAAPS would fit within their schools well 
and that the framework, particularly the training and ongo- 
ing coaching, would address existing barriers including 
lack of adequate training and trained staff. No changes to 
the MAAPS modules were recommended by stakeholders. 
Stakeholders uniformly commented on the collaborative 
nature of MAAPS as a positive feature, and so we modified 
a manual we developed for coaches to provide more guid- 
ance around collaborative work with educators and families. 
For example, we included material on how to develop and 
maintain rapport, provide support to teachers, and how to 
give performance feedback in an evidence-based and sup- 
portive manner. 


Phase Il: Pilot Randomized Controlled Trial 


Because there are currently no comprehensive modular 
interventions with empirical support in school settings, our 
primary goal was to assess educator perceptions of the fea- 
sibility and sustainability of MAAPS after they had experi- 
ence implementing the intervention. Feasibility outcomes 
are critical in new interventions, as they provide the fol- 
lowing information: (1) Proof-of-concept for the ability to 
recruit and retain target participants; (2) Evidence support- 
ing the ability to implement procedures with fidelity; (3) 
Data on social validity of interventions, which could inform 
intervention refinements; and (4) Quality of data collected. 
In combination, the data gathered from a feasibility study are 
essential for supporting the rationale for a larger-scale effi- 
cacy trial. As described earlier, we used the RE-AIM model 
(Kessler et al. 2012) to evaluate feasibility by comparing the 
intervention and control groups. Indicators, data collection 
methods and measures used to operationalize RE-AIM are 
in Table | and are described in the next section. Because 
our primary question had to do with overall feasibility of 
MAAPS and the sensitivity of our measures to detect change 
in the RE-AIM framework, we elected to conduct an under- 
powered RCT (smaller sample size) to determine whether it 


Dimension Indicator(s) Data collection method/measure 
methods and measures used 
to operationalize ‘A each, Reach Students receiving the intervention at each Demographic data 
effectiveness, adoption, school at exit 
implementation, and : 
‘ Effectiveness Student outcomes DD-GAS 
maintenance (RE-AIM) ; ; ; 
dimensions Adoption Ratings of contextual fit at exit URP-IR 
Implementation Fidelity ratings Coach self-assessment 
Maintenance Fidelity at exit as compared to mid-year Fidelity checklist 
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is reasonable and feasible to conduct a full trial and whether 
the components of MAAPS were effective. 


Methods 
Participants and Setting 


The study was conducted in Florida, Massachusetts, and 
New York. The study was approved by the Institutional 
Review Board (IRB), and University of Rochester served as 
the IRB of record. We used a hierarchical recruitment pro- 
cess, that moved through permissions from district admin- 
istration, to building principals, to classroom educators, and 
finally to parents and students (when able to do so). Consent 
to participate was provided by teachers and parents. When 
appropriate, students provided assent. 

Students were recruited from 14 schools (4 in Florida, 
4 in Massachusetts, and 6 in New York). Using National 
Center for Educational Statistics (USDE 2017) categori- 
zation, 71% of participating schools were suburban, 14% 
urban, and 14% rural. Ninety-three percent of the schools 
were eligible to provide free and reduced fee lunches, and 
57% were Title I schools. The average school size was 620 
students (range 508 to 748). Student teacher ratio averaged 
14:1 (range 13:1 to 14:1). 

Student inclusion criteria were as follows: (1) placement 
in grades, K-8 (2) receipt of individual educational plan 
(IEP) services under the educational category of Autism 
in their district; and (3) confirmation of ASD diagnostic 
criteria via the Autism Diagnostic Observation Schedule, 
2nd Edition (ADOS-2; Lord et al. 2012) as administered 
by a research-certified diagnostician. The ADOS is consid- 
ered the gold standard of autism diagnostics and is a semi- 
structured assessment that includes standardized activities 
with a combination of highly structured activities and inter- 
actions and less structured interactions. Teacher criteria 
included (1) having an eligible student in their classroom 
whose parents provided consent and who, when appropri- 
ate, provided assent to participate; and (2) identification as 
a “lead” teacher (i.e., the teacher with whom the consented 
student spent the bulk of his/her school day). To characterize 
the sample, data were collected on student IQ from school 
records, if completed within a 1-year period, otherwise pro- 
ject staff administered an appropriate and psychometrically 
valid assessment of IQ. 

A total of 35 students and 39 educators were screened 
for participation. Of potential student participants, 2 did not 
meet diagnostic criteria for ASD, 4 were excluded because 
educators declined to participate, and one student was 
excluded due to a change in school placement after con- 
sent but prior to randomization. Of the 39 educators who 
were screened, nine consented to participate; however, their 
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selected student did not participate, and two declined to par- 
ticipate (one was taking family/medical leave and the other 
switched to a new classroom). Thus, a total of 28 students 
and 28 educators participated. Demographic information for 
all participants is in Table 2. Of note, although the mean IQ 
for both groups was in the average range, there was diversity 
of IQ in both groups (range 47-139). 

Because we were evaluating the feasibility of the MAAPS 
intervention, project staff served as MAAPS coaches. 
Coaches held a Master’s or Doctoral degree in Behavior 
Analysis, School Psychology, or Clinical Psychology and 
had backgrounds in behavior analysis and school-based 
interventions for students with ASD. A single coach worked 
with each student and his/her teacher and team; each coach 
worked with multiple teams. Prior to serving as coaches, 
each attended a 3-h training on implementation of MAAPS, 
reviewed the coaching manual, and the goals and process for 
each of the 27 modules (see Table 4). When a module was 
selected by the team, the coach received behavioral skills 
training on the module. This included didactic instruction, 
modeling, and role-plays, and continued until the coach 
was able to implement all features of the module with 100% 
fidelity during role-plays. The duration of training ranged 
between 30 min and 1| h per module. The duration of training 
varied somewhat per coach but ranged between 30 min and 
1 h per module. completed after module selection and prior 
to the next scheduled meeting. 


Measures 


As noted earlier, we used the RE-AIM model (Kessler et al. 
2012) to evaluate feasibility and potential for sustainabil- 
ity by comparing the intervention and control groups. The 
schedule of measures is in Table 3. 

Reach is a measure of the number of individuals who 
participated in MAAPS as well as the extent to which par- 
ticipants represented our target population (students with 
ASD and their teachers). We evaluated our reach using 
demographic data on students, teachers, and schools. These 
data were collected at screening via demographic surveys 
(parents, educators). 

Effectiveness is a measure of the efficacy of MAAPS and 
involves assessing both positive outcomes and any negative 
or unintended consequences. For the latter, we developed a 
comprehensive system for reporting unintended outcomes 
and protocol deviations, however none was reported by par- 
ticipants or observed by our research team. We measured 
effectiveness in several different ways. To assess changes in 
overall functioning we used the Developmental Disability- 
Clinical Global Assessment Scale (DD-CGAS; Wagner et al. 
2007). The DD-CGAS is a rating scale organized in deciles 
(range 1-100 where | is extreme and pervasive impairment 
and 100 is superior functioning) and is used to assess global 


Journal of Autism and Developmental Disorders 


Table 2 Participant characteristics 


MAAPS ETAU p-value 
(n= 14) (n= 14) 
Students 
Gender 0.307 
Male 10 (71%) 11 (79%) 
Female 4 (29%) 3 (21%) 
Age (years): M(SD) 8.07 (2.0)* 8.14 (2.1)* 0.926 
1Q: M(SD) 90.4 (24.3)* 84.5 (25.0)? 0.540 
ADOS Comparison Score 7.7 (1.8)* 7.6 (1.5)* 0.947 
Race 0.296 
African-American 3 (21%) 0 (0%) 
American Indian/Alaskan Native 0 (0%) 1(7%) 
Asian 2 (14%) 0 (0%) 
Caucasian 9 (64%) 12 (86%) 
Multi-racial 0 (0%) 1(7%) 
Ethnicity 0.684 
Hispanic/Latino/Spanish 1(7%) 2 (14%) 
Not Hispanic/Latino/Spanish 12 (86%) 10 (71%) 
Did not disclose 1(7%) 2 (14%) 
Classroom placement 0.184 
General education 0 (0%) 2 (14%) 
General education with some special education 5 (36%) 7 (50%) 
General special education 1(7%) 2 (14%) 
DD-specific special education 8 (57%) 3 (21%) 
Attending Title 1 School 6 (43%) 6 (43%) 0.481 
MAAPS ETAU p-value 
(n= 14) (n= 14) 
Teachers 
Gender 
Male 0 (0%) 0 (0%) 
Female 14 (100%) 14 (100%) 
Race 0.222 
African-American 1 (8%) 0 (0%) 
American Indian/Alaskan Native 1 (8%) 0 (0%) 
Asian 1 (8%) 0 (0%) 
Caucasian 11 (84%) 14 (100%) 
Multi-racial 0 (0%) 0 (0%) 
Ethnicity 0.480 
Hispanic/Latino/Spanish 0 (0%) 2 (14%) 
Not Hispanic/Latino/Spanish 12 (92%) 11 (84%) 
Did not disclose 1 (8%) 0 (0%) 
Highest level of education 0.157 
Bachelor’s degree 4 (1%) 2 (14%) 
Graduate degree 7 (54%) 12 (86%) 
Other 2 (14%) 0 (0%) 
Did not disclose 1(7%) 0 (0%) 


DD developmental disabilities, ETAU enhanced treatment as usual, MAAPS modular approach for autism programs in schools 


“Standard deviation 
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Table 3 Schedule of measures 


Measure Assessment timepoint 


Mid-inter- Exit 
vention 


Pre-inter- 
vention 


Screen 


ADOS-2 + 
SB-5 + 
SSIS-RS + + 
ABAS-3 + + 
CYBOCS-ASD* + 45 + 
DD-CGAS?* + + + 
TNTB* + + + 
AET + + + 
URP-IR + 
Coach Quality + + 


ADOS-2 Autism Diagnostic Observation Schedule, Second Edition, 
SB-5 Stanford Binet Intelligence Scales, Fifth Edition, SS7S-RS Social 
Skills Improvement System Rating Scales, ABAS-3 Adaptive Behav- 
ior Assessment System, Third Edition, CYBOCS-ASD Children’s 
Yale-Brown Obsessive Compulsive Scale for children with autism 
spectrum disorder,, DD-CGAS Developmental Disabilities Clinical 
Global Assessment Scale, TNTB Teacher Nominated Target Behav- 
iors, AET Academic Engaged Time, URP-IR Usage Rating Profile- 
Intervention (Revised) 


*Measures completed by independent evaluators blind to treatment 
condition 


Optional assessment timepoint 


functioning in treatment studies with children with ASD. 
The DD-CGAS has excellent inter-rater reliability (0.79) 
and temporal stability (0.86), shows convergent validity 
with other measures of functioning (0.50), and is sensitive 
to treatment effects (Wagner et al. 2007). Blinded raters 
completed the DD-CGAS using scores on other measures at 
baseline, at midpoint, and at the end of the year (exit, June). 
In addition to the primary outcomes (i.e., DD-CGAS), we 
identified several secondary outcomes. 

We assessed changes in core deficits of ASD using two 
measures, the Social Skills Improvement System-Rating 
Scales (SSIS-RS; Gresham and Elliott 2008) and the Chil- 
dren’s Yale-Brown Obsessive Compulsive Scale, ASD Ver- 
sion (CY-BOCS-ASD Scahill et al. 2006). The SSIS is an 
informant-rated questionnaire that includes items related to 
two broad domains: Social Skills and Problem Behavior. 
Informants rate the individual on the frequency of vari- 
ous behaviors using a 4-point Likert scale (Never, Seldom, 
Often, Always). The SSIS-RS has good median scale (0.90) 
and test-retest reliability (0.82—0.87; Gresham et al. 2011), 
and it correlates with other well-established measures, such 
as the Vineland and the BASC (Gresham et al. 2010). In this 
study, teachers completed the SSIS-RS at baseline, midpoint, 
and exit. Domain scores were used as secondary outcome 
measures, in addition to informing independent evaluator 
ratings on the DD-CGAS. The CYBOCS-ASD is a modified 
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version of the CY-BOCS originally developed for use in 
children with Obsessive—Compulsive Disorder and later 
adapted for children with ASD (Scahill et al. 2014) to rate 
the current severity of repetitive behavior. The CYBOCS- 
ASD is a semi-structured scale completed by trained asses- 
sors (project staff). A given participant’s repetitive behav- 
iors are rated on five dimensions: Time Spent, Interference, 
Distress, Resistance, and Control on a 5-point scale from 
0 (least symptomatic) to 4 (most symptomatic), yielding a 
Total score from 0 to 20. It has established internal con- 
sistency (0.86) and reliability (0.90; Wu et al. 2014) and 
is a standard outcome measure for assessing restricted and 
repetitive behavior in intervention studies of ASD (Scahill 
et al. 2014). The CY-BOCS ASD was used as a secondary 
outcome measure and to inform independent evaluator rat- 
ings for the DD-CGAS. 

We measured changes in the following features often 
associated with a diagnosis of ASD: challenging behavior, 
adaptive behavior, academic engagement. We assessed chal- 
lenging behavior using an adapted form of Goal Attainment 
Scaling (Ruble et al. 2012; Schlosser 2004) that we called 
Teacher-Nominated Target Behavior (TNTB). Studies using 
Goal Attainment Scaling have assessed progress toward IEP 
goals but because student IEP goals did not necessarily align 
with core and associated features of ASD, we instead asked 
teachers to identify three priority outcomes for the student 
at baseline. An independent evaluator helped the teacher 
develop operational definitions of the outcomes prior to the 
start of the study. Then, the evaluator reviewed each skill 
and definition with the teacher at midpoint and exit, and 
an improvement rating was assigned (from | = Very much 
improved to 7= Very much worse). 

To assess changes in adaptive behavior we used the Adap- 
tive Behavior Assessment System, Second Edition (ABAS- 
2 Harrison and Oakland 2003)—Self-Direction Subscale. 
Teachers provided ratings on the Self-Direction Subscale 
of the ABAS-2 at baseline and exit to assess changes in the 
student’s independent academic and social behaviors in the 
classroom. Items were rated on a 4-point Likert scale from 0 
(not able) to 3 (always). The ABAS has acceptable internal 
consistency (0.98) and construct validity (0.78-0.93) for the 
subscales). 

Finally, as a measure of engagement with academic activ- 
ities, we used Academic Engaged Time (AET). The AET, 
part of the multi-gated Systematic Screening of Behavior 
Disorders (SSBD; Walker and Severson 1990), is a direct 
observational measure that uses a stop-watch to record the 
duration of time a student is actively engaged during inde- 
pendent instructional time. Each student was observed by a 
trained rater for two 15—20 min sessions which were aver- 
aged at each measurement interval (baseline, mid-point, 
exit). Training procedures included raters getting a didac- 
tic training on observational procedures used by the AET, 
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examples and non-examples of AET definitions, practice 
using the AET with videos of students in classroom sessions, 
and feedback from project PIs. All raters were required to 
demonstrate 0.90 interobserver agreement prior to collecting 
the data from the participants. The AET has been used in 
multiple research studies and has shown excellent interrater 
reliability (Walker et al. 2009). 

Measures of adoption assess the number and representa- 
tiveness of the individuals who participated and factors that 
may affect widespread adoption. Within the MAAPS group, 
we assessed this via the Usage Rating Profile-Intervention 
(Revised) (URP-IR; Chafouleas et al. 2009). The URP-IR 
is a 29-item, 6-point Likert rating (Strongly disagree to 
Strongly agree) with four factors: acceptability, understand- 
ing (which may reflect likelihood of continued use), feasi- 
bility, and systems-support for the intervention. The four 
factors of the URP-IR were derived from a study of 254 
educators (Chafouleas et al. 2009). The URP-IR was com- 
pleted at the end of the school year. 

Implementation is an indicator of fidelity of each element 
of the intervention. For MAAPS that included fidelity of 
coaching and coaching quality, and teacher use of interven- 
tions developed via MAAPS modules. 


Fidelity of Coaching 


First, coaches used a fidelity checklist to self-assess dose of 
coaching and adherence to the MAAPS process (i.e., coach 
fidelity). The checklist was completed after each coaching 
session. Second, project staff scored 30% of team-meetings 
using a fidelity checklist of key features to be covered during 
team meetings. To facilitate this process, all team meetings 
were audio-recorded and at least 30% (range 30-70% scored 
for IOA) were randomly selected for coding by an independ- 
ent evaluator. 


Teacher Fidelity and Quality 


Second, coaches monitored fidelity of implementation of 
MAAPS modules by teachers (i.e., teacher fidelity) using 
a 10-item checklist. The checklist had four content areas: 
General/Organizational Components (e.g., intervention 
implemented when it should have been, educator arranged 
environment or had materials needed), Teaching/Ante- 
cedent Strategies (specific antecedent strategies such as 
reviewing rules), Prompting Strategies (specific stimulus 
and/or response prompts as indicated in the student’s inter- 
vention), and Consequence Strategies (specific strategies 
to follow desired and/or undesired behavior). Each time a 
coach assessed fidelity of implementation of a module by a 
teacher, the coach also assessed quality of implementation. 
The quality rating indicated how well teachers delivered the 
intervention to students. Quality was assessed by three items 


including: Rapport and Responsiveness (e.g., attentive to 
student, high ratio of positive to negative statements), Com- 
munication (e.g., developmentally and child-appropriate 
tone and volume, clear and specific instructions, appropri- 
ate nonverbal communication), and Flexibility (e.g., imple- 
mented intervention fluently) on the same 3-point scale. 
Ratings were assigned across individual items for adher- 
ence (O=not completed to 3=full adherence) and quality 
(0=seldom demonstrates the skill to 3=always demon- 
strates the skill). Means were expressed as a percentage of 
points earned divided by total possible points. 


Coaching Quality Scale 


We assessed teacher-reported quality of coaching delivery 
using a 10-item checklist with items rated on a 4-point scale 
ranging from never to always. Items on this scale evaluated 
teacher impressions of coach interactions such as the extent 
to which they agreed on goals, whether the coach communi- 
cated effectively, was approachable, and whether the teacher 
trusted the coach. Coaching quality was evaluated at mid- 
point and again at the end of the year. 


Randomization and Condition Descriptions 


Participant flow is documented in Fig. 1. A cluster rand- 
omized design was implemented in this study. Participating 
schools from 3 sites were equally assigned to either MAAPS 
or enhanced treatment as usual (ETAU) group. The rand- 
omization sequence was generated by the project statisti- 
cian and was stratified by site. Randomization occurred once 
consent from participating teachers and students’ parents 
was obtained. 


MAAPS 


The MAAPS program is a framework for implementation 
of evidence-based treatments. Interventions in MAAPs 
were identified based on systematic reviews of the literature 
(National Autism Center 2015; Wong et al. 2015), that used 
published and agreed upon criteria for defining a practice 
as evidence-based. Although the initial aim was to include 
only interventions with evidence supporting their effective- 
ness in school settings, there were insufficient interventions 
that met this standard (Anderson et al. 2017); therefore we 
included all interventions that met criteria established by 
Wong et al., and the National Autism Center as evidence- 
based regardless of where implementation occurred. Inter- 
ventions in MAAPS are organized into modules around the 
core and associated features of ASD as depicted in Table 4 
Each module was designed to be implemented independently 
of others, as a standalone module, although multiple mod- 
ules could be implemented simultaneously. 
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Screened 


Enrollment 


n= 35 students 
n = 39 educators 


Not meeting inclusion criteria (n=17) 
Students whose educator declined (n = 4) 
Students did not meet ADOS criteria (n =2) 
Students no longer in active teacher’s class 
(n=1) 
Educator without consented student (n=9) 
Declined to participate (n=2 educators) 


Randomized 
n= 28 students 
n= 27 educators 


Allocation 


MAAPS 
n= 14 students 
n= 14 educators 


Analysis 


ETAU 
n= 14 students 
n= 13 educators 


Withdrawals n = 3 


n= 11 dyads 


Fig. 1 MAAPS consort diagram 


Module selection for a student occurred during an initial 
MAAPS meeting attended by the student’s teacher, parent 
or care provider, and any other stakeholders identified as 
important by the parent and teacher (e.g., speech therapist). 
During this meeting, coaches used a semi-structured inter- 
view to help team members identify goals for a student and 
select appropriate module(s) based on a student’s specific 
strengths and deficits that would directly address the goals. 
If a team selected multiple modules, the coach helped them 
prioritize up to three modules, and to set a tentative date 
to begin each module, only one module was taught at a 
time. When the team was ready to begin another module, 
the coach met with the teacher and provided training on the 
next selected module. 
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Lost to follow-up n= 1 


n= 13 dyads 


Each intervention module included all of the materials a 
coach would need to teach an educator how to design, initi- 
ate, and implement an intervention including: (a) an over- 
view of the conceptual basis and empirical support for the 
intervention in the module, (b) an outline for the coach to 
follow in meeting with an educator to fully develop the inter- 
vention and train the teacher in its use, (c) sample materials, 
and (d) vignettes illustrating implementation of the mod- 
ule. Once an intervention module was selected, the coach 
met with the teacher to explain the main components of the 
intervention and to determine key features of implementa- 
tion. For example, if the intervention focused on increas- 
ing requesting, the coach would help the teacher identify 
activities or items the student enjoyed, determine how each 


Journal of Autism and Developmental Disorders 


Table 4 MAAPS modules by 
core or associated feature of 


autem addrmaced Foundational skills 


Requesting 

Basic communication 

Peer partner and engagement 
Basic play skills 


Advanced communication Skills 


Nonverbal communication 
Recognizing social cues 
Conversation 

Sustaining social interactions 
Group games 

Group activities 

Dramatic play 

Developing friendships 

Peer networks 


would be requested (e.g., by saying the name of the item or 
an approximation of the name), and when and how teach- 
ing would occur (e.g., how many times per day, by whom, 
where). Prior to implementation, the coach used behavioral 
skills training (e.g., Parsons et al. 2012) to help the teacher 
learn the intervention. When implementation began, the 
coach conducted classroom observations at least weekly 
to assess fidelity of implementation and provide feedback 
based on teacher reflection and observations to enhance 
implementation. The MAAPS team continued to meet at 
least monthly (face-to-face or via email as desired by the 
team) to monitor outcomes and plan next steps, including 
implementing subsequent selected modules. 


Enhanced Treatment as Usual 


Teachers in schools randomized to the control group 
received a book guiding implementation of a school-based 
intervention (Smith 2011). In addition, the schools in the 
control group were offered an array of training topics aligned 
with the MAAPS modules from which they could select up 
to 12 h of didactic training. Training presentations were 
1-3 h in length and covered key features of interventions 
associated with the selected topic; didactics also focused on 
implementation of procedures in a school setting. No other 
supports were provided. 


Analysis Plan 
Baseline group differences were assessed across demographic 


variables, as well as primary and secondary outcomes using 
t-tests (for continuous variables) or Chi-square tests (for 


Social communication and interaction 


Cognitive/academic difficulties 
Motivation and work completion 
Visual cues 
Reinforcement systems 
Environmental modifications 
Opportunities to respond 
Learning new skills 
Instructional support 
Learning strategies 
Peer tutoring 


Restricted and repetitive behavior 

Difficulty with unexpected changes 
Planning for the unplanned 
Schedules 

Repetitive behaviors 
Automatically reinforced SIB 
Motor stereotypy 
Vocal stereotypy 
Sensory differences 

Restricted interests 
Increasing variability 
Engagement 
Rituals 


nominal variables). All outcome analyses were conducted fol- 
lowing the intention-to-treat principle. For the primary out- 
comes, linear mixed effects models were fitted with repeated 
DD-CGAS scores as dependent variables, and treatment group 
as the independent variable. Based upon previous literature, 
analyses were controlled for baseline measures, student IQ, 
ethnicity, classroom type, and whether schools were desig- 
nated as Title 1. To evaluate how the outcome at all assess- 
ment points, including baseline, changed over time for each 
group, the interaction between treatment and time was tested. 
Specified contrasts were tested to assess both the within-group 
changes and the between group-differences at each time point. 
Effect sizes were calculated as Cohen’s d for paired samples, 
to account for change from baseline. Site effects were investi- 
gated by comparing the DD-CGAS outcomes across three sites 
and calculating the Intraclass Correlation Coefficient (ICC). 
Identical procedures were conducted for secondary outcomes. 
Missing data and missingness patterns were examined to deter- 
mine if the subjects with missing data differed significantly 
from the remaining sample in terms of subject characteristics. 
All missing data were determined to be missing completely at 
random. Inference-based Linear Mixed Modeling is valid for 
situations that include non-informative (i.e., randomly) miss- 
ing data or when missingness does not depend on the value of 
the unobserved outcome. 
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Results 
Baseline Analyses 


Both groups were compared at baseline on multiple child and 
teacher demographic characteristics, as well as on the outcome 
measures. No significant baseline group differences were 
observed, with the exception of higher impairment ratings for 
the Self-Care Domain of the DD-CGAS for the MAAPS group 
(p =0.048) (see Tables 2 and 5). At baseline, IQ was positively 
correlated with DD-CGAS total score (p <0.001) and nega- 
tively related to impairment across the following DD-CGAS 
subdomains: Communication (p < 0.001), School/ Academic 
(p <0.001), and Self-Care (p =0.007), but not Social Behavior 
(p=0.61). More time spent by the student in a special educa- 
tion setting was positively correlated to impairment on three 
DD-CGAS subdomains (i.e., Communication; p=0.004; 
School/ Academic; p=0.01; and Self-Care; p=0.05) and 
negatively related to DD-CGAS total score (p =0.03). With 
respect to school characteristics, placement in a Title 1 school 
was positively related to impairment for the Communication 
(p =0.04) subdomain and negatively related to DD-CGAS 
total score (p=0.03). 


Reach 


We assessed REACH by comparing the number of students for 
whom a MAAPS team formed to the number of students still 
participating at exit (see Fig. 1). Our study periods included 
recruitment (October 2017—February 2018), baseline (Janu- 
ary—February 2018), and exit (May—June, 2018). With respect 
to recruitment and retention, we enrolled 74% of participants 
screened, including teachers and students. Across sites, 85% 
of students in schools randomized to MAAPS completed the 
study (12 of the 14 students) and 93% of students in schools 
assigned to control completed the study (13 of 14 students). 
All withdrawals occurred due to either a change in student 
placement (one student each in MAAPS and control) or to 
loss of a teacher (one student in MAAPS whose teacher took 
family/medical leave). 


Effectiveness 

Although we were underpowered to detect meaningful change 
and to confidently extrapolate from the findings, efficacy anal- 
yses were completed with the goals of identifying initial trends 
in child outcomes. 


Preliminary Child Outcomes 


Preliminary efficacy was explored on our primary child-level 
outcome, overall school functioning, with the DD-CGAS 
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(see Table 5). Students in the MAAPS group showed gradual 
increases in DD-CGAS scores from baseline (M=51.5) to 
midpoint (M = 54) and then at exit (M@=61). When control- 
ling for baseline scores, the MAAPS group demonstrated a 
significant main effect of time, with an increase in the mean 
DD-CGAS total score at exit (p < 0.001) and with a large 
effect size (d= 0.94). In addition to statistically significant 
change, the increased total score in the MAAPS group rep- 
resents a clinically significant improvement in functioning 
from “moderate” impairment to “slight” impairment. In 
contrast, the ETAU group did not show change in function- 
ing across timepoints (i.e., M=57.5 at baseline, M=58.5 at 
midpoint, M=60 at exit; p=0.22), with scores remaining 
stable. 

A group by time analysis did not reveal significant inter- 
action effects for either change from baseline to midpoint 
(p =0.44) or for change from baseline to exit (p=0.67) for 
the DD-CGAS Total Score and for the subdomain impair- 
ment ratings (Table 5). However, for DD-CGAS Total Score 
there was group by time interaction for improvement from 
midpoint to exit that favored the MAAPS group over the 
ETAU group (Difference in Least Squared Means = 3.19; 
p=0.01). Further, visual analysis (see Fig. 2; DD-CGAS 
Total Score) demonstrates that the pattern of results is sug- 
gestive of a group by time interaction at exit for the overall 
DD-CGAS and for the individual subscales. In the context 
of these favorable trends for MAAPS, the lack of significant 
findings is likely resultant from low power to detect change. 

In addition to DD-CGAS total score, we evaluated the 
groups across the four domains (i.e., social behavior, com- 
munication, school/academic, and self-care; see Table 5 and 
Figs. 3, 4, 5 and 6). With respect to raw means, students 
in the MAAPS group were rated as having greater impair- 
ment than the ETAU group at baseline across all domains. 
For communication (Fig. 3), social behavior (Fig. 4), and 
school/academic (Fig. 5), the MAAPS group improved over 
time, such that impairment was lower at exit, as compared to 
ETAU, although not significant (p=0.06). Within the com- 
munication and school/academic domains, MAAPS had 
caught up to ETAU by midpoint. However, when controlling 
for baseline score, these improvements were not statistically 
significant. In contrast to the overall trends towards improve- 
ment in MAAPS, the ETAU group’s impairment ratings 
were fairly stable (i.e., school/academic, communication) 
or decreased slightly (i.e., social behavior) over time. There 
were no changes in either group over time on the self-care 
subdomain (Fig. 6), with the MAAPS group rated as having 
more impairment at baseline, midpoint, and exit. 


Secondary Child Outcomes 


A summary of within-group analysis can be found in 
Table 6. Results at mid- and exit indicated that the 
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Table 5 Means and standard deviations of both groups for primary and secondary outcome measures 


Scale MAAPS ETAU p value* 
M SD M SD 
DD-CGAS total 0.69 
Baseline 51.50 17.69 57.50 16.50 
Mid-point 54.00 20.71 58.54 15.38 
Exit 60.91 16.33 59.92 14.20 
Social behavior domain 0.45 
Baseline 2.57 0.85 2.43 0.65 
Mid-point 2.50 1.00 2.15 0.69 
Exit 1.82 0.87 2.08 0.76 
School/academic domain 0.90 
Baseline 2.36 1.15 1.93 1.14 
Mid-point 2.00 0.85 2.08 1.11 
Exit 1.72 0.64 1.92 1.04 
Communication domain 0.50 
Baseline 2.21 1.05 1.86 1.23 
Mid-point 2.00 0.85 2.00 1.15 
Exit 1.72 1.00 2.00 1.08 
Self-care domain 0.13 
Baseline 1.64 1.00 0.86 0.95 
Mid-point 1.67 1.23 0.85 0.98 
Exit 1.45 1.04 0.85 0.99 
SSIS social skills 0.04* 
Baseline 72.46 14.84 73.85 12.32 
Exit 86.33 11.10 78.85 12.28 
SSIS problem behavior 0.75 
Baseline 121.31 18.45 118.77 11.14 
Exit 113.25 13.60 113.69 35.69 
Scale MAAPS ETAU p value 
M SD M SD 
CY-BOCS total score 0.48 
Baseline 14.00 3.14 14.15 3.63 
Exit 10.55 3.88 12.15 3.76 
ABAS self-direction 0.54 
Baseline 29.90 8.55 29.20 11.61 
Exit 35.77 11.04 33.00 11.70 
Academic engaged time 0.66 
Baseline 67.14 16.04 72.57 13.93 
Mid-point 73.00 14.59 64.31 22.21 
Exit 72.08 24.88 65.08 25.50 
Odds ratio p value 
Teacher nominated target behaviors 0.017% 


N= 28 at baseline and 24 at exit 


p valuebetween group differences at exit, ABAS Adaptive Behavior Assessment Scales, CY-BOCS Children’s Yale-Brown Obsessive-Compul- 
sive Scale, DD-CGAS Developmental Disabilities modification of the Children’s Global Assessment Scale, ETAUEnhanced Treatment as Usual, 


MAAPS Modular Approach to Autism Programs in Schools, SS/S Social Skills Improvement System 


*p <0.05 
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Fig. 2 DD-CGAS total scores at baseline, midpoint, and exit 
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Fig. 3 DD-CGAS communication ratings at baseline, midpoint, and 
exit 
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Fig.4 DD-CGAS social behavior ratings at baseline, midpoint, and 
exit 


MAAPS group experienced an increase in percentage of 
academic engagement on the AET from baseline (67.1) 
to exit (72.1), whereas there was a decrease in academic 
engagement in the ETAU group (from 72.6 to 65.1) that 
approached significance at midpoint (p = 0.07; see Fig. 7). 
However, the group by time interaction was not significant 
at midpoint or exit (Table 5 DD-CGAS). On the SSIS, 
MAABS significantly outperformed ETAU at exit on the 
Social Skills scale (p =0.04; see Fig. 8), and the treatment 
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Fig.5 DD-CGAS school/academic ratings at baseline, midpoint, and 
exit 
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Fig.6 DD-CGAS self-care ratings at baseline, midpoint, and exit 


Table 6 Within-group analyses for MAAPS and ETAU from baseline 
to exit 


Measure MAAPS ETAU 

ES p ES P 
DD-CGAS total score 0.94 <0.001 0.37 0.22 
Social behavior domain — 1.04 0.45 —0.16 00.90 
School/academic domain —0.24 0.86 -041 0.79 
Communication domain —0.30 0.87 ns ns 
Self-care domain na 0.92 na ns 
SSIS social skills 1.21 0.002 0.45 0.12 
SSIS problem behavior 0.78 0.02 —0.21 0.46 
CY-BOCS total score 0.28 0.38 0.11 0.71 
ABAS self-direction 0.21 0.12 00.46 0.12 
Academic engaged time —0.02 0.73 0.46 0.10 


Statistically significant values are given in bold 


ABAS Adaptive Behavior Assessment Scales, CY-BOCS Children’s 
Yale-Brown Obsessive Compulsive Scale, DD-CGAS Developmental 
Disabilities Modification of the Children’s Global Assessment Scale, 
ESEffect Size (Cohen’s d), ETAUnhanced Treatment as Usual, LSM- 
Diff Difference in Least Square Means, naChange scores too close to 
0 to calculate effect size, ns Not significant, where the p-value was too 
close to 1 to be interpretable, MAAPS Modular Approach to Autism 
Programs in Schools 
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Fig. 8 SSIS social skills standard scores at baseline and exit 


group also demonstrated a significant increase in Social 
Skills score from baseline (72.5) to exit (86.3; p=0.002) 
and with a large effect size (d= 1.21), whereas change in 
ETAU (73.8 at baseline and 78.8 at exit) was not signifi- 
cant. Both groups demonstrated a decrease in the SSIS 
Problem Behavior scale (from 121.3 to 113.3 in MAAPS 
and 118.8 to 113.7 in ETAU), which was only significant 
in the MAAPS group (p=0.02); and there was no notable 
interaction (Fig. 9). However, the effect size in MAAPS 
was moderate-to-large (0.78) as compared to a small effect 
size (0.21) for ETAU. With respect to repetitive behav- 
iors, students experienced a non-significant decrease in 
repetitive and ritualistic behaviors on the CY-BOCS-in 
the MAAPS group (from 14 to 10.5) and the ETAU group 
(from 14.1 to 12.2), with no indication of a group by time 
interaction (Fig. 10). Ratings on the Self-Direction sub- 
scale of the ABAS indicated no significant changes in 
scores obtained from baseline to exit in the MAAPS group 
(5.7) or in the ETAU group (3.8; Fig. 11). Improvement 
ratings from the Teacher Nominated Target Behaviors 
(TNTB) indicated that students in MAAPS were signifi- 
cantly more likely to have any identified behaviors rated 
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Fig.9 SSIS problem behavior standard scores at baseline and exit 
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Fig. 11 ABAS self-direction subscale scores at baseline and exit 


as improved, as compared to those in ETAU (OR3.61; 
p=0.017). 


Adoption 
Mean factor and item scores on the URP-IR ranged from | 


(strongly disagree) to 6 (strongly agree). Factor and item- 
level scores are in Table 7. Factor-level scores on four of 
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the domains—Acceptability = 5.2, Understanding = 5.1, 
Feasibility =5.5, and System Climate = 5.3—were closer to 
the maximum score of 6. These factors indicate an overall 
positive perception of MAAPS in the areas assessed and 
specifically suggest that educators found MAAPS to be very 
acceptable, that they understood the intervention framework, 
that it was feasible, and that systems features in their school 
and district were supportive. On the Home-School Collabo- 
ration factor, educators rated parent involvement as not nec- 
essary for the success of MAAPS (a rating of 3.7 indicates 
“Slightly Disagree”) which suggests the intervention could 


Table 7 Usage rating profile 
scores 


Factor 


Acceptability 


This intervention is an effective choice for addressing a variety of problems 
The intervention is a fair way to handle the child’s behavior problem 
I would not be interested in implementing this intervention* 
I would have positive attitudes about implementing this intervention 
This intervention is a good way to handle the child’s behavior problem 
I would implement this intervention with a good deal of enthusiasm 
This intervention would not be disruptive to other students 
I would be committed to carrying out this intervention 


The intervention procedures easily fit in with my current practices 


Understanding 


I understand how to use this intervention 
I am knowledgeable about the intervention procedures 


I understand the procedures of this intervention 


Feasibility 


I would be able to allocate my time to implement this intervention 

The total time required to implement the intervention procedures would be manageable 
Preparation of materials needed for this intervention would be minimal 

Material resources needed for this intervention are reasonable 

This intervention is too complex to carry out accurately* 


The amount of time required for record keeping would be reasonable 


Home school collaboration 


A positive home-school relationship is needed to implement this intervention 
Parental collaboration is required in order to use this intervention 


Regular home-school communication is needed to implement intervention procedures 


System climate 


My administrator would be supportive of my use of this intervention 

Use of this intervention would be consistent with the mission of my school 
Implementation of this intervention is well matched to what is expected in my job 
These intervention procedures are consistent with the way things are done in my system 


My work environment is conducive to implementation of an intervention like this one 


System support? 


I would need additional resources to carry out this intervention 
I would need consultative support to implement this intervention 


I would require additional professional development in order to implement this intervention 
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be useful even for students whose parents are not able to be 
involved with school-based services. This removes a poten- 
tial barrier to widespread adoption, as some parents may be 
unwilling or unable to participate in a school-based inter- 
vention. Finally, elevated scores in the System Support fac- 
tor (M=5.2) indicate that more supports, such as coaching, 
resources, and training, are needed for long-term sustainabil- 
ity of the intervention than what is currently available. Most 
item scores within each factor are consistent with the overall 
ratings, with the exception of slightly elevated agreement on 
one item (“I would not be interested in implementing this 


Mean (SD) 


5.2 (0.3) 
5.3 (0.6) 
5.5 (0.7) 
3.7 (1.7) 
5.3 (0.6) 
5.2 (0.6) 
5.4 (0.7) 
5.3 (0.6) 
5.3 (0.6) 
5.2 (0.8) 
5.1 (0.1) 
5.6 (0.7) 
5.5 (0.7) 
5.5 (0.7) 
5.5 (0.3) 
5.0 (1.1) 
5.3 (0.6) 
5.3 (0.8) 
5.3 (0.8) 
3.4 (1.9) 
5.2 (0.7) 
3.7 (2) 

3.8 (1.6) 
3.6 (1.6) 
2.5 (1.6) 
5.3 (0.2) 
5.5 (0.7) 
5.3 (0.6) 
5.3 (0.8) 
5.1 (1.0) 
5.3 (0.8) 
5.2 (0.6) 
3.4 (1.1) 
2.6 (1.5) 
3.4 (1.7) 


“These items were reverse scored 


®As per developer instructions, items in this factor were reverse scored so that a score of 6.0 reflected high 
support, aligning scoring with other factors 
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intervention’) within the Acceptability domain, indicating 
some ambivalence. The high standard deviation for this item 
shows that this perspective was not consistent across educa- 
tors. Further, it is not clear whether raters indicating less 
interest in implementing did so because they found MAAPS 
to be either not useful or feasible or rather that it was no 
longer needed because students who received MAAPS no 
longer required intervention. 


Implementation 


We assessed fidelity of implementation of modules by teach- 
ers and teacher impression of the quality (value) of coaching, 
as shown in Table 8. Overall, high teacher fidelity was main- 
tained across the 7 coaching schools, which ranged from 
90.1 to 98.1, and scores were very high (mean at or above 
95%). On the coaching quality scale, teachers provided a 
mean rating of 3.8 (0.07) out of a 4-point Likert scale (see 
Table 8 Teacher Fidelity). Questions included items such 
as “The coach and I agree on the most important goals for 
intervention for my student.” High ratings were consistent 
across schools, with no average scores dropping below 3.0. 


Coach fidelity was also very strong for adherence (rang- 
ing from 93.9 to 100) and quality (ranging from 96.5 to 
100; see Table 9). All inter-rater agreement scores fell at 
95% agreement or higher. 


Maintenance 


Given that this trial was focused on feasibility and pre- 
liminary efficacy, no long-term follow-up was planned and 
therefore we cannot comment yet on intervention main- 
tenance. As discussed earlier, scores on the URP-15 (see 
Table 7) indicated that educators perceive there is support 
for longer-term maintenance and that they would be com- 
mitted to carrying out the intervention in the future. In 
the context of these early promising results, a larger-scale 
trial that includes systematic evaluation of maintenance is 
a logical next step. 


Table 8 Fidelity and quality 


Coach-rated teacher imple- 


Coach-rated teacher implemen- Teacher-rated coach- 


ratings mentation fidelity tation quality ing quality 
Mean (%) Range Mean (%) Range Mean Range 
School 1 96.3 81-100 100 100-100 3.8 3.44.0 
School 2 94.4 75-100 95.6 67-100 3.9 3.64.0 
School 3 n/a® n/a n/a n/a 3.9 3.8-3.9 
School 4 90.1 78-100 100 100-100 4.0 4.0-4.0 
School 5 n/a? n/a n/a n/a 3.0 3.0-3.0 
School 6 93.9 90-97 100 100-100 3.0 3.0-4.0 
School 7 98.1 96-100 100 100-100 4.0 4.0-4.0 
Overall mean 94.6 99.1 3.8 
*Data not collected due to scheduling conflicts 
>Participant at site withdrew before collected 
Table 9 Coach implementation Site Coach self-rated Coach self-rated Independent evaluator- Independent 
fidelity of MAAPS procedures implementation adher- implementation quality rated adherenceIOA __ evaluator-rated 
by school ence quality IOA 
Mean (%) Range Mean (%) Range Mean (%) Mean (%) 
School1 100 90-100 100 100-100 94.5 100 
School2 97.8 87-100 99.3 91-100 92.3 100 
School3 94.2 94-100 95.8 91-100 94.2 95.8 
School 4 96.5 95-100 100 100-100 97.9 100 
School5 93.9 94-100 100 100-100 n/a* n/a 
School6 100 100-100 100 100-100 100 100 
School7 100 100-100 100 100-100 100 100 


*Participant at site withdrew before completed 
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Discussion 


The purpose of this study was to evaluate the feasibility of 
further evaluation of MAAPS. We began by recruiting feed- 
back from experts in the field and stakeholders and used 
this information to make modifications to MAAPS. We then 
conducted a randomized controlled trial to assess feasibility 
in a more in-depth manner and used the RE-AIM framework 
to guide our research analysis. Overall, results supported 
the potential feasibility and efficacy of MAAPS within the 
RE-AIM framework. With respect to reach, we successfully 
enrolled a majority of individuals screened for the study and 
the few individuals enrolled who did not complete the study 
dropped out due to extenuating circumstances (change in 
district placement, medical leave). 

Our data also support the effectiveness of MAAPS, 
as students who participated in the MAAPS group sig- 
nificantly improved their overall functioning, the primary 
outcome, as measured by the DD-CGAS compared to the 
students who participated in the ETAU group. Students 
in the MAAPS group also significantly improved in sec- 
ondary outcomes, including social skills and classroom 
behaviors identified as important by teachers, compared 
to their ETAU counterparts. Teachers found MAAPS to 
be highly acceptable, useful, and effective, and strong 
educator buy-in, as indicated by low rates of withdrawal 
from MAAPS and high levels of participation, may have 
contributed to data showing that teachers implemented the 
MAAPS interventions with high fidelity. 

The likelihood of adoption is a key consideration in 
novel interventions because, if the intervention is unlikely 
to be taken up by end-users, then there is little need for 
further research. Our preliminary data suggest that edu- 
cators overall found MAAPS to be acceptable, easy to 
understand, feasible, and a good contextual fit—all of 
which indicate a strong likelihood of adoption. Educators 
did note that MAAPS would require systems-level sup- 
port from schools and districts. This suggests that, prior to 
implementing MAAPS, efforts should be made to secure 
buy-in at the district level to ensure resources will be avail- 
able for initial and sustained implementation. 

Implementation has to do with whether (and how well) 
and intervention is actually implemented as designed. We 
observed high levels of both fidelity and quality of imple- 
mentation, suggesting that educators were willing and 
able to implement MAAPS with the support of coaches. 
Finally, we conducted an assessment of the potential for 
maintenance of MAAPs, by asking educators to self- 
report on several variables that could influence sustained 
adoption and maintenance over time. Educators indicated 
willingness to implement MAAPs over time and that they 
would have support within their school or district to do so. 
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Taken together, results from this feasibility study, sug- 
gest several important directions for future research. More 
specifically, our findings suggest that MAAPS is feasible 
in schools, that it resulted in positive outcomes, and that 
it has a high likelihood of being adopted and sustained 
by educators. MAAPS has unique features that may have 
contributed to the results. First, it is a comprehensive 
intervention that uses a modular approach to give teams 
multiple options about the selection of specific focused 
interventions, the quantity of interventions to be imple- 
mented, and the timing of when and how the interven- 
tions will be scheduled for implementation throughout 
the school year. This flexibility is key as students with 
ASD are very diverse in their presentation of diagnostic 
characteristics and associated difficulties and strengths, 
creating challenges for educators with limited resources 
to sufficiently ameliorate concerns and prioritize how and 
where to intervene. This diversity of behavioral presen- 
tation, mentioned often by the focus group participants, 
overwhelms educators in attempting to meet student needs, 
particularly in the absence of structured frameworks to 
guide application of EBIs. 

A second unique feature of MAAPS is the embed- 
ded active coaching process. Extant literature identifies 
coaching as a necessary component to help teachers with 
implementation and generalization of new interventions 
for sustainability (e.g., Fixsen et al. 2009; Noell et al. 
2005). Coaching processes that include performance feed- 
back based on direct observations of teacher performance 
have been shown to be particularly powerful in increas- 
ing teacher implementation fidelity (Solomon et al. 2012). 
This was clearly the case in our stakeholder groups, as the 
coaching component was enthusiastically endorsed by all 
of the focus groups as one of the key factors that would 
increase the contextual fit of MAAPS in teacher class- 
rooms and help support teachers to implement EBIs. 

The MAAPS active coaching process blends two coach- 
ing models, practice-based coaching (Snyder et al. 2015) 
and behavior skills training (Hogan et al. 2015), that is 
provided in an ongoing, iterative cycle in which both per- 
formance feedback and teacher reflection were utilized 
to increase implementation fidelity to criterion. As seen 
in the fidelity data outcomes, most of the teachers in the 
MAAPS group achieved high implementation fidelity, and 
coaches also showed fidelity to implementation of MAAPS 
procedures. This bidirectionality of the coaching process 
is critical, as the higher quality and process fidelity of the 
coaches will result in higher teacher fidelity and perceived 
social validity of the intervention (Stormont et al. 2015). 
The URP-IR results appeared to confirm this bidirectional- 
ity as teachers found both the MAAPS framework and the 
coaching value to be high. 
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Some limitations exist with the current study. First, the study 
was designed for a small N; thus, it was underpowered. A 
more rigorous randomized controlled trial with a larger N 
needs to be conducted to further examine the effectiveness 
of MAAPS and explore what child and teacher factors may 
moderate intervention effectiveness. To this point, it will be 
critical for such evaluations to include student participants 
with diversity in presentation of cognitive levels and charac- 
teristics to help identify whether the MAAPS framework has 
the sufficient flexibility to be applicable for any student with 
ASD or is more effective for students with specific needs. 
Further, due to the small N, not all of the MAAPS modules 
were selected and implemented; thus, it is unknown whether 
some modules may be more feasibly implemented or more 
effective than other modules. 

A second limitation is that coaching was provided by 
research staff who were highly trained. We chose to use 
research staff as coaches so we could evaluate the potential 
of MAAPS under “ideal conditions.” Coaching is evidence- 
based; however, research is needed to develop models for 
training educators to serve as coaches in MAAPS or other 
coach-driven interventions. Additionally, more research is 
necessary to determine the minimal dosage of coaching that 
can impact high teacher implementation fidelity. Both the 
expert reviewers and the focus group participants expressed 
concerns that implementation of MAAPS may require more 
resources (e.g., time and skills) than were available in typi- 
cal schools. This issue would need to be resolved in order 
for MAAPS to be both implemented and sustained as an 
adopted framework in school settings. 

To our knowledge, this is the first study that explored the 
feasibility of a modular approach to support implementa- 
tion of EBIs for students with ASD in authentic schools. 
Further, this is one of the first feasibility studies that used 
the RE-AIM framework for structuring the research plan and 
evaluating the outcomes. Our findings suggest that modu- 
lar approaches may be a viable programmatic option for 
implementation in authentic school environments to meet 
the needs of students with ASD. Finally, using a framework 
such as RE-AIM can aid the field in determining the viability 
and sustainability of future intervention research. 

As described in more detail above, all procedures involv- 
ing human participants were in accordance with the ethical 
standards of the University of Rochester Medical Center, 
which served as the IRB of record and with the 1964 Hel- 
sinki declaration and its later amendments or comparable 
ethical standards. 
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